Skip to content

feat(strategy): rrqr-nystrom -- redundancy-avoiding landmark selection for decaying-spectrum data#226

Draft
RenzoMXD wants to merge 1 commit into
zeokin:mainfrom
RenzoMXD:feat/nystrom-three-way
Draft

feat(strategy): rrqr-nystrom -- redundancy-avoiding landmark selection for decaying-spectrum data#226
RenzoMXD wants to merge 1 commit into
zeokin:mainfrom
RenzoMXD:feat/nystrom-three-way

Conversation

@RenzoMXD

@RenzoMXD RenzoMXD commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

PR kind

  • fix
  • feat

DRAFT -- GPU scorecard pending. Replaces this PR's third attempt (block-krylov). No GPU available right now for this content; opening as a draft to avoid the missing-scorecard auto-close.

Why the previous content is gone

block-krylov was repeat-tested (5 fresh-seed GPU runs, pinned decaying-spectrum regime --data-rank 256 --rank-m 256) and turned out to be a regression versus the retired nystrom-iter attempt on both axes at once:

transform accuracy latency vs exact (~222ms)
block-krylov 0.843 - 0.864 552-577ms (~2.5x slower)
nystrom-iter (retired) 0.875 - 0.891 370-403ms (~1.7x slower)
nystrom 0.808 - 0.833 182-207ms (faster than exact)

Splitting a fixed per-part width budget across Krylov depth traded away direct landmark breadth for GEMM-refined columns and lost more than it gained -- the second consecutive confirmation that spending the fixed M budget on extra GEMMs against the full n x n operand burns latency faster than it buys accuracy at this scale, no matter how the GEMMs are arranged.

Checked the premise numerically before writing a third implementation

Before touching GPU time again, two cheap NumPy checks (no GPU needed):

  1. Leverage-score-weighted landmark sampling -- ruled out. Computed true leverage scores of this project's own decaying-spectrum fill directly: they come out essentially uniform (std/mean ~0.08, matching a random subspace). The fill's k^-alpha decay is applied to an abstract rank-direction axis that a dense Gaussian mix then spreads evenly across every physical column -- there is no per-column importance signal to weight by on this synthetic data. Any cheap leverage-score proxy (e.g. raw column norm) would just be chasing sampling noise, not real structure.
  2. Redundancy-avoiding selection -- confirmed real. Uniform-random landmark draws are wasteful in a different way: by chance they can include near-parallel (redundant) columns. Oversampling a small candidate pool (oversample * w landmarks, still just a memory gather) and greedily picking the least-redundant subset via column-pivoted Gram-Schmidt (RRQR-style) beat plain uniform sampling's column-space recovery in 8/8 trials (mean rel-err 0.120 vs 0.132, tight std bands, no overlap).

Summary (new content)

New transform rrqr-nystrom:

  • same 3-way space split {col(A), row(A), row(B)} as every prior attempt on this PR (#156's redundancy proof still holds)
  • oversamples oversample * w (default oversample=4) uniform landmark columns/rows per part -- still a pure memory gather, same mechanism as nystrom
  • greedily selects the least-redundant w of them via column-pivoted Gram-Schmidt: repeatedly pick the remaining candidate with the largest residual norm after projecting out everything already chosen

Crucially, the pivoted selection runs on the small oversampled pool, not a GEMM against the full n x n operand -- cost is O(n w²), not O(n² w), roughly two orders of magnitude cheaper at n=8192 than nystrom-iter/block-krylov's GEMM-refinement approach. This directly targets the failure mode that sank both previous attempts on this PR, rather than trying a third variant of the same GEMM-refinement idea.

Verified so far (CPU, numpy only -- not a claim of GPU dominance)

test_rrqr_nystrom_beats_plain_nystrom_on_decaying_spectrum reproduces the pre-implementation numeric check as an automated test: pivoted selection from an oversampled pool spans col(A) measurably better than an equal-size uniform draw on decaying-spectrum data. 9 tests total: registration, input validation (including graceful handling of oversample=1), orthonormality, seed determinism, exact low-rank recovery, and honest (non-zero) basis_flops.

CPU-safe validation

$ uv run python -m strategy.smoke        # 3/3 transforms pass
$ uv run --extra test python -m pytest tests/ strategy/tests/ eval/tests/ -q
321 passed, 24 skipped

GPU Result (required for feat PRs only)

Pending -- will attach before marking ready for review. Plan: same pinned decaying-spectrum regime used for both prior honest negative results on this PR (--fill decaying-spectrum --data-rank 256 --rank-m 256), comparing rrqr-nystrom against nystrom (isolates the selection mechanism's contribution) and rsvd, 5+ fresh seeds -- not a single run.

Scope

strategy/transforms.py + strategy/README.md + strategy/tests/test_rrqr_nystrom.py only (Open -- "the main event" zone per CONTRIBUTING.md); new transform, no API change, no protected-path edits.

Checklist

  • CPU-safe validation passed
  • My commits do not include Co-authored-by footers for coding agents
  • GPU scorecard attached (pending -- draft)

@github-actions github-actions Bot added area:strategy Smart strategies / transforms (strategy/) area:tests Correctness gates and test suites (tests/) status:needs-review Awaiting maintainer review status:needs-scorecard Feature / strategy PR is missing the reproducible eval scorecard type:feature Feature / improvement PR that must carry a GPU scorecard type:strategy New smart strategy / transform proposal labels Jul 13, 2026
@RenzoMXD
RenzoMXD marked this pull request as ready for review July 14, 2026 21:37
@github-actions github-actions Bot added status:queued-gpu Feat/strategy PR passed non-GPU gates and is queued for the next batched GPU evaluation and removed status:needs-scorecard Feature / strategy PR is missing the reproducible eval scorecard labels Jul 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Gate chain passed. This PR is queued for the next batched GPU evaluation window.

@RenzoMXD
RenzoMXD force-pushed the feat/nystrom-three-way branch from 56d7c4f to a7dd9dd Compare July 15, 2026 02:56
@github-actions github-actions Bot added the area:docs README, CONTRIBUTING, BENCHMARKS and other docs label Jul 15, 2026
@RenzoMXD RenzoMXD changed the title feat(strategy): nystrom captures 3 subspaces, not 4 (drop redundant col(B) landmark block) feat(strategy): nystrom-iter -- landmark-seeded power iteration for decaying-spectrum data Jul 15, 2026
@github-actions github-actions Bot added the status:needs-scorecard Feature / strategy PR is missing the reproducible eval scorecard label Jul 15, 2026
@RenzoMXD
RenzoMXD marked this pull request as draft July 15, 2026 02:57
@github-actions github-actions Bot removed the status:needs-scorecard Feature / strategy PR is missing the reproducible eval scorecard label Jul 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Gate chain passed. This PR is queued for the next batched GPU evaluation window.

@RenzoMXD
RenzoMXD force-pushed the feat/nystrom-three-way branch from a7dd9dd to 9615bcf Compare July 15, 2026 04:09
@github-actions github-actions Bot added the status:needs-scorecard Feature / strategy PR is missing the reproducible eval scorecard label Jul 15, 2026
@RenzoMXD RenzoMXD changed the title feat(strategy): nystrom-iter -- landmark-seeded power iteration for decaying-spectrum data feat(strategy): block-krylov -- landmark-seeded block Krylov subspace for decaying-spectrum data Jul 15, 2026
…n for decaying-spectrum data

Replaces this PR's block-krylov content. Repeat GPU testing (5 fresh-seed
runs, --data-rank 256 --rank-m 256) showed block-krylov was a regression
versus the retired nystrom-iter attempt on both axes at once: worse accuracy
(0.843-0.864 vs nystrom-iter's 0.875-0.891) and worse latency (~2.5x slower
than exact vs ~1.7x). Splitting a fixed per-part width budget across Krylov
depth traded away direct landmark breadth for GEMM-refined columns and lost
more than it gained -- the second consecutive confirmation that spending the
fixed M budget on extra GEMMs against the full n x n operand burns latency
faster than it buys accuracy at this scale.

Before implementing a third attempt, checked the actual premise numerically
first this time. A "leverage-score nystrom" idea (weight landmark draws by
column importance) was ruled out by computing true leverage scores of this
project's decaying-spectrum test fill directly: they come out essentially
uniform (std/mean ~0.08, matching a random subspace), because the fill's
k^-alpha decay is applied to an abstract rank-direction axis that a dense
Gaussian mix then spreads evenly across every physical column -- there is no
per-column importance signal to weight by. What a second numeric check found
instead: uniform-random landmark draws are wasteful in a different way --
they can by chance include redundant (near-parallel) columns. Oversampling a
small candidate pool and greedily selecting the least-redundant subset via
column-pivoted Gram-Schmidt (RRQR-style) beat plain uniform sampling's
column-space recovery in 8/8 trials (mean rel-err 0.120 vs 0.132).

This only needs a small oversampled candidate pool, not a GEMM against the
full operand, so cost is O(n w^2) rather than O(n^2 w) -- roughly two orders
of magnitude cheaper at n=8192 than every GEMM-refinement approach tried on
this PR so far, directly avoiding the failure mode that sank both of them.
@RenzoMXD
RenzoMXD force-pushed the feat/nystrom-three-way branch from 9615bcf to 42d9d33 Compare July 15, 2026 05:56
@RenzoMXD RenzoMXD changed the title feat(strategy): block-krylov -- landmark-seeded block Krylov subspace for decaying-spectrum data feat(strategy): rrqr-nystrom -- redundancy-avoiding landmark selection for decaying-spectrum data Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs README, CONTRIBUTING, BENCHMARKS and other docs area:strategy Smart strategies / transforms (strategy/) area:tests Correctness gates and test suites (tests/) status:needs-review Awaiting maintainer review status:needs-scorecard Feature / strategy PR is missing the reproducible eval scorecard status:queued-gpu Feat/strategy PR passed non-GPU gates and is queued for the next batched GPU evaluation type:feature Feature / improvement PR that must carry a GPU scorecard type:strategy New smart strategy / transform proposal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant